{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "934b74e1",
   "metadata": {},
   "source": [
    "# CLIQ-ai’21 DLG4NLP Tutorial Demo: Semantic Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4346fd68",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767c3b2e",
   "metadata": {},
   "source": [
    "In this tutorial demo, we will use the Graph4NLP library to build a GNN-based semantic parsing model. The model consists of \n",
    "- graph construction module (e.g., node embedding based dynamic graph)\n",
    "- graph embedding module (e.g., Bi-Sep GAT)\n",
    "- prediction module (e.g., RNN decoder with attention, copy and coverage mechanisms)\n",
    "\n",
    "We will use the built-in Graph2Seq model APIs to build the model, and evaluate it on the Jobs dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac004ea7",
   "metadata": {},
   "source": [
    "### Environment setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de88301",
   "metadata": {},
   "source": [
    "Please follow the instructions [here](https://github.com/graph4ai/graph4nlp_demo#environment-setup) to set up the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "086ca306",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using backend: pytorch\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import yaml\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "from graph4nlp.pytorch.datasets.jobs import JobsDataset\n",
    "from graph4nlp.pytorch.models.graph2seq import Graph2Seq\n",
    "from graph4nlp.pytorch.models.graph2seq_loss import Graph2SeqLoss\n",
    "from graph4nlp.pytorch.modules.graph_construction import *\n",
    "from graph4nlp.pytorch.modules.evaluation.base import EvaluationMetricBase\n",
    "from graph4nlp.pytorch.modules.utils.config_utils import update_values\n",
    "from graph4nlp.pytorch.modules.utils.copy_utils import prepare_ext_vocab\n",
    "from graph4nlp.pytorch.modules.config import get_basic_args"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf1b0d6b",
   "metadata": {},
   "source": [
    "### Build the model handler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03b7543",
   "metadata": {},
   "source": [
    "Let's build a model handler which will do a bunch of things including setting up dataloader, model, optimizer, evaluation metrics, train/val/test loops, and so on.\n",
    "\n",
    "We will call the Graph2Seq model API which implements a GNN-based encoder and LSTM-based decoder.\n",
    "\n",
    "When setting up the dataloader, users will need to call the dataset API which will preprocess the data, e.g., calling the graph construction module, building the vocabulary, tensorizing the data. Users will need to specify the graph construction type when calling the dataset API.\n",
    "\n",
    "Users can build their customized dataset APIs by inheriting our low-level dataset APIs. We provide low-level dataset APIs to support various scenarios (e.g., `Text2Label`, `Sequence2Labeling`, `Text2Text`, `Text2Tree`, `DoubleText2Text`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b1637d23",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Jobs:\n",
    "    def __init__(self, opt):\n",
    "        super(Jobs, self).__init__()\n",
    "        self.opt = opt\n",
    "        self.use_copy = self.opt[\"decoder_args\"][\"rnn_decoder_share\"][\"use_copy\"]\n",
    "        self.use_coverage = self.opt[\"decoder_args\"][\"rnn_decoder_share\"][\"use_coverage\"]\n",
    "        self._build_device(self.opt)\n",
    "        self._build_dataloader()\n",
    "        self._build_model()\n",
    "        self._build_loss_function()\n",
    "        self._build_optimizer()\n",
    "        self._build_evaluation()\n",
    "\n",
    "    def _build_device(self, opt):\n",
    "        seed = opt[\"seed\"]\n",
    "        np.random.seed(seed)\n",
    "        if opt[\"use_gpu\"] != 0 and torch.cuda.is_available():\n",
    "            print('[ Using CUDA ]')\n",
    "            torch.manual_seed(seed)\n",
    "            torch.cuda.manual_seed_all(seed)\n",
    "            from torch.backends import cudnn\n",
    "            cudnn.benchmark = True\n",
    "            device = torch.device('cuda' if opt[\"gpu\"] < 0 else 'cuda:%d' % opt[\"gpu\"])\n",
    "        else:\n",
    "            print('[ Using CPU ]')\n",
    "            device = torch.device('cpu')\n",
    "        self.device = device\n",
    "\n",
    "    def _build_dataloader(self):\n",
    "        dataset = JobsDataset(\n",
    "            root_dir=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\"root_dir\"],\n",
    "            pretrained_word_emb_name=self.opt[\"pretrained_word_emb_name\"],\n",
    "            pretrained_word_emb_cache_dir=self.opt[\"pretrained_word_emb_cache_dir\"],\n",
    "            merge_strategy=self.opt[\"graph_construction_args\"][\"graph_construction_private\"][\n",
    "                \"merge_strategy\"\n",
    "            ],\n",
    "            edge_strategy=self.opt[\"graph_construction_args\"][\"graph_construction_private\"][\n",
    "                \"edge_strategy\"\n",
    "            ],\n",
    "            seed=self.opt[\"seed\"],\n",
    "            word_emb_size=self.opt[\"word_emb_size\"],\n",
    "            share_vocab=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\n",
    "                \"share_vocab\"\n",
    "            ],\n",
    "            graph_name=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\n",
    "                \"graph_name\"\n",
    "            ],\n",
    "            dynamic_init_graph_name=self.opt[\"graph_construction_args\"][\n",
    "                \"graph_construction_private\"\n",
    "            ].get(\"dynamic_init_graph_type\", None),\n",
    "            topology_subdir=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\n",
    "                \"topology_subdir\"\n",
    "            ],\n",
    "            thread_number=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\n",
    "                \"thread_number\"\n",
    "            ],\n",
    "            port=self.opt[\"graph_construction_args\"][\"graph_construction_share\"][\"port\"],\n",
    "        )\n",
    "\n",
    "        self.train_dataloader = DataLoader(\n",
    "            dataset.train,\n",
    "            batch_size=self.opt[\"batch_size\"],\n",
    "            shuffle=True,\n",
    "            num_workers=self.opt[\"num_workers\"],\n",
    "            collate_fn=dataset.collate_fn,\n",
    "        )\n",
    "        self.test_dataloader = DataLoader(\n",
    "            dataset.test,\n",
    "            batch_size=self.opt[\"batch_size\"],\n",
    "            shuffle=False,\n",
    "            num_workers=self.opt[\"num_workers\"],\n",
    "            collate_fn=dataset.collate_fn,\n",
    "        )\n",
    "        self.vocab = dataset.vocab_model\n",
    "\n",
    "    def _build_model(self):\n",
    "        # Call the Graph2Seq model API\n",
    "        self.model = Graph2Seq.from_args(self.opt, self.vocab).to(self.device)\n",
    "\n",
    "    def _build_loss_function(self):\n",
    "        # Call the Graph2Seq loss API\n",
    "        self.loss = Graph2SeqLoss(ignore_index=self.vocab.out_word_vocab.PAD,\n",
    "                                  use_coverage=self.use_coverage, coverage_weight=0.3)\n",
    "    def _build_optimizer(self):\n",
    "        parameters = [p for p in self.model.parameters() if p.requires_grad]\n",
    "        self.optimizer = optim.Adam(parameters, lr=self.opt[\"learning_rate\"])\n",
    "\n",
    "    def _build_evaluation(self):\n",
    "        self.metrics = [ExpressionAccuracy()]\n",
    "\n",
    "    def train(self):\n",
    "        max_score = -1\n",
    "        self._best_epoch = -1\n",
    "        for epoch in range(self.opt[\"epochs\"]):\n",
    "            self.model.train()\n",
    "            self.train_epoch(epoch, split=\"train\")\n",
    "            self._adjust_lr(epoch)\n",
    "            if epoch >= 0:\n",
    "                score = self.evaluate(split=\"test\")\n",
    "                if score >= max_score:\n",
    "                    print(\"Best model saved, epoch {}\".format(epoch))\n",
    "                    self.save_checkpoint(\"best.pth\")\n",
    "                    self._best_epoch = epoch\n",
    "                max_score = max(max_score, score)\n",
    "            if self._stop_condition(epoch):\n",
    "                break\n",
    "        return max_score\n",
    "\n",
    "    def _stop_condition(self, epoch, patience=20):\n",
    "        return epoch > patience + self._best_epoch\n",
    "\n",
    "    def _adjust_lr(self, epoch):\n",
    "        def set_lr(optimizer, decay_factor):\n",
    "            for group in optimizer.param_groups:\n",
    "                group['lr'] = group['lr'] * decay_factor\n",
    "\n",
    "        epoch_diff = epoch - self.opt[\"lr_start_decay_epoch\"]\n",
    "        if epoch_diff >= 0 and epoch_diff % self.opt[\"lr_decay_per_epoch\"] == 0:\n",
    "            if self.opt[\"learning_rate\"] > self.opt[\"min_lr\"]:\n",
    "                set_lr(self.optimizer, self.opt[\"lr_decay_rate\"])\n",
    "                self.opt[\"learning_rate\"] = self.opt[\"learning_rate\"] * self.opt[\"lr_decay_rate\"]\n",
    "                print(\"Learning rate adjusted: {:.5f}\".format(self.opt[\"learning_rate\"]))\n",
    "\n",
    "    def train_epoch(self, epoch, split=\"train\"):\n",
    "        assert split in [\"train\"]\n",
    "        print(\"Start training in split {}, Epoch: {}\".format(split, epoch))\n",
    "        loss_collect = []\n",
    "        dataloader = self.train_dataloader\n",
    "        step_all_train = len(dataloader)\n",
    "        for step, data in enumerate(dataloader):\n",
    "            graph, tgt, gt_str = data[\"graph_data\"], data[\"tgt_seq\"], data[\"output_str\"]\n",
    "            graph = graph.to(self.device)\n",
    "            tgt = tgt.to(self.device)\n",
    "            oov_dict = None\n",
    "            if self.use_copy:\n",
    "                oov_dict, tgt = prepare_ext_vocab(graph, self.vocab, gt_str=gt_str, device=self.device)\n",
    "\n",
    "            prob, enc_attn_weights, coverage_vectors = self.model(graph, tgt, oov_dict=oov_dict)\n",
    "            loss = self.loss(logits=prob, label=tgt, enc_attn_weights=enc_attn_weights,\n",
    "                             coverage_vectors=coverage_vectors)\n",
    "            loss_collect.append(loss.item())\n",
    "            if step % self.opt[\"loss_display_step\"] == 0 and step != 0:\n",
    "                print(\"Epoch {}: [{} / {}] loss: {:.3f}\".format(epoch, step, step_all_train,\n",
    "                                                                           np.mean(loss_collect)))\n",
    "                loss_collect = []\n",
    "            self.optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            self.optimizer.step()\n",
    "\n",
    "    def evaluate(self, split=\"val\"):\n",
    "        self.model.eval()\n",
    "        pred_collect = []\n",
    "        gt_collect = []\n",
    "        assert split in [\"val\", \"test\"]\n",
    "        dataloader = self.val_dataloader if split == \"val\" else self.test_dataloader\n",
    "        for data in dataloader:\n",
    "            graph, tgt, gt_str = data[\"graph_data\"], data[\"tgt_seq\"], data[\"output_str\"]\n",
    "            graph = graph.to(self.device)\n",
    "            if self.use_copy:\n",
    "                oov_dict = prepare_ext_vocab(batch_graph=graph, vocab=self.vocab, device=self.device)\n",
    "                ref_dict = oov_dict\n",
    "            else:\n",
    "                oov_dict = None\n",
    "                ref_dict = self.vocab.out_word_vocab\n",
    "\n",
    "            prob, _, _ = self.model(graph, oov_dict=oov_dict)\n",
    "            pred = prob.argmax(dim=-1)\n",
    "\n",
    "            pred_str = wordid2str(pred.detach().cpu(), ref_dict)\n",
    "            pred_collect.extend(pred_str)\n",
    "            gt_collect.extend(gt_str)\n",
    "\n",
    "        score = self.metrics[0].calculate_scores(ground_truth=gt_collect, predict=pred_collect)\n",
    "        print(\"Evaluation accuracy in `{}` split: {:.3f}\".format(split, score))\n",
    "        return score\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def translate(self):\n",
    "        self.model.eval()\n",
    "\n",
    "        pred_collect = []\n",
    "        gt_collect = []\n",
    "        dataloader = self.test_dataloader\n",
    "        for data in dataloader:\n",
    "            graph, tgt, gt_str = data[\"graph_data\"], data[\"tgt_seq\"], data[\"output_str\"]\n",
    "            graph = graph.to(self.device)\n",
    "            if self.use_copy:\n",
    "                oov_dict = prepare_ext_vocab(batch_graph=graph, vocab=self.vocab, device=self.device)\n",
    "                ref_dict = oov_dict\n",
    "            else:\n",
    "                oov_dict = None\n",
    "                ref_dict = self.vocab.out_word_vocab\n",
    "\n",
    "            pred = self.model.translate(batch_graph=graph, oov_dict=oov_dict, beam_size=4, topk=1)\n",
    "\n",
    "            pred_ids = pred[:, 0, :]  # we just use the top-1\n",
    "\n",
    "            pred_str = wordid2str(pred_ids.detach().cpu(), ref_dict)\n",
    "\n",
    "            pred_collect.extend(pred_str)\n",
    "            gt_collect.extend(gt_str)\n",
    "\n",
    "        score = self.metrics[0].calculate_scores(ground_truth=gt_collect, predict=pred_collect)\n",
    "        return score\n",
    "\n",
    "    def load_checkpoint(self, checkpoint_name):\n",
    "        checkpoint_path = os.path.join(self.opt[\"checkpoint_save_path\"], checkpoint_name)\n",
    "        self.model.load_state_dict(torch.load(checkpoint_path))\n",
    "\n",
    "    def save_checkpoint(self, checkpoint_name):\n",
    "        checkpoint_path = os.path.join(self.opt[\"checkpoint_save_path\"], checkpoint_name)\n",
    "        if not os.path.exists(self.opt[\"checkpoint_save_path\"]):\n",
    "            os.makedirs(self.opt[\"checkpoint_save_path\"], exist_ok=True)\n",
    "        torch.save(self.model.state_dict(), checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dac6e553",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExpressionAccuracy(EvaluationMetricBase):\n",
    "    def __init__(self):\n",
    "        super(ExpressionAccuracy, self).__init__()\n",
    "\n",
    "    def calculate_scores(self, ground_truth, predict):\n",
    "        correct = 0\n",
    "        assert len(ground_truth) == len(predict)\n",
    "        for gt, pred in zip(ground_truth, predict):\n",
    "            if gt == pred:\n",
    "                correct += 1.\n",
    "        return correct / len(ground_truth)\n",
    "\n",
    "def wordid2str(word_ids, vocab):\n",
    "    ret = []\n",
    "    assert len(word_ids.shape) == 2, print(word_ids.shape)\n",
    "    for i in range(word_ids.shape[0]):\n",
    "        id_list = word_ids[i, :]\n",
    "        ret_inst = []\n",
    "        for j in range(id_list.shape[0]):\n",
    "            if id_list[j] == vocab.EOS:\n",
    "                break\n",
    "            token = vocab.getWord(id_list[j])\n",
    "            ret_inst.append(token)\n",
    "        ret.append(\" \".join(ret_inst))\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ebf198e",
   "metadata": {},
   "source": [
    "### Set up the config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a702636a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# config setup\n",
    "config_file = '../config/jobs/gat_bi_sep_dynamic_node_emb.yaml'\n",
    "config = yaml.load(open(config_file, 'r'), Loader=yaml.FullLoader)\n",
    "\n",
    "opt = get_basic_args(graph_construction_name=config[\"graph_construction_name\"],\n",
    "                              graph_embedding_name=config[\"graph_embedding_name\"],\n",
    "                              decoder_name=config[\"decoder_name\"])\n",
    "update_values(to_args=opt, from_args_list=[config, config[\"other_args\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "985bc54a",
   "metadata": {},
   "source": [
    "### Run the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0cb15c2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ Using CPU ]\n",
      "Building vocabs...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".vector_cache/glove.6B.zip: 862MB [06:06, 2.35MB/s]                                            \n",
      "100%|███████████████████████████████████████████████▉| 399999/400000 [00:55<00:00, 7231.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretrained word embeddings hit ratio: 0.717948717948718\n",
      "Using pretrained word embeddings\n",
      "[ Initialized word embeddings: (195, 300) ]\n",
      "Saving vocab model to ../data/jobs/processed/node_emb_graph/vocab.pt\n",
      "Start training in split train, Epoch: 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/hugo/opt/anaconda3/envs/g4l_nov2021b/lib/python3.7/site-packages/torch-1.10.0-py3.7-macosx-10.9-x86_64.egg/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling\n",
      "  warnings.warn('User provided device_type of \\'cuda\\', but CUDA is not available. Disabling')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: [10 / 21] loss: 4.008\n",
      "Epoch 0: [20 / 21] loss: 2.515\n",
      "Evaluation accuracy in `test` split: 0.000\n",
      "Best model saved, epoch 0\n",
      "Start training in split train, Epoch: 1\n",
      "Epoch 1: [10 / 21] loss: 1.810\n",
      "Epoch 1: [20 / 21] loss: 1.421\n",
      "Evaluation accuracy in `test` split: 0.000\n",
      "Best model saved, epoch 1\n",
      "Start training in split train, Epoch: 2\n",
      "Epoch 2: [10 / 21] loss: 1.187\n",
      "Epoch 2: [20 / 21] loss: 1.132\n",
      "Evaluation accuracy in `test` split: 0.000\n",
      "Best model saved, epoch 2\n",
      "Start training in split train, Epoch: 3\n",
      "Epoch 3: [10 / 21] loss: 1.024\n",
      "Epoch 3: [20 / 21] loss: 0.991\n",
      "Evaluation accuracy in `test` split: 0.171\n",
      "Best model saved, epoch 3\n",
      "Start training in split train, Epoch: 4\n",
      "Epoch 4: [10 / 21] loss: 0.933\n",
      "Epoch 4: [20 / 21] loss: 0.880\n",
      "Evaluation accuracy in `test` split: 0.129\n",
      "Start training in split train, Epoch: 5\n",
      "Epoch 5: [10 / 21] loss: 0.836\n",
      "Epoch 5: [20 / 21] loss: 0.864\n",
      "Evaluation accuracy in `test` split: 0.107\n",
      "Start training in split train, Epoch: 6\n",
      "Epoch 6: [10 / 21] loss: 0.804\n",
      "Epoch 6: [20 / 21] loss: 0.778\n",
      "Evaluation accuracy in `test` split: 0.229\n",
      "Best model saved, epoch 6\n",
      "Start training in split train, Epoch: 7\n",
      "Epoch 7: [10 / 21] loss: 0.743\n",
      "Epoch 7: [20 / 21] loss: 0.724\n",
      "Evaluation accuracy in `test` split: 0.336\n",
      "Best model saved, epoch 7\n",
      "Start training in split train, Epoch: 8\n",
      "Epoch 8: [10 / 21] loss: 0.669\n",
      "Epoch 8: [20 / 21] loss: 0.681\n",
      "Evaluation accuracy in `test` split: 0.400\n",
      "Best model saved, epoch 8\n",
      "Start training in split train, Epoch: 9\n",
      "Epoch 9: [10 / 21] loss: 0.643\n",
      "Epoch 9: [20 / 21] loss: 0.648\n",
      "Evaluation accuracy in `test` split: 0.479\n",
      "Best model saved, epoch 9\n",
      "Start training in split train, Epoch: 10\n",
      "Epoch 10: [10 / 21] loss: 0.606\n",
      "Epoch 10: [20 / 21] loss: 0.611\n",
      "Evaluation accuracy in `test` split: 0.521\n",
      "Best model saved, epoch 10\n",
      "Start training in split train, Epoch: 11\n",
      "Epoch 11: [10 / 21] loss: 0.572\n",
      "Epoch 11: [20 / 21] loss: 0.552\n",
      "Evaluation accuracy in `test` split: 0.536\n",
      "Best model saved, epoch 11\n",
      "Start training in split train, Epoch: 12\n",
      "Epoch 12: [10 / 21] loss: 0.552\n",
      "Epoch 12: [20 / 21] loss: 0.508\n",
      "Evaluation accuracy in `test` split: 0.564\n",
      "Best model saved, epoch 12\n",
      "Start training in split train, Epoch: 13\n",
      "Epoch 13: [10 / 21] loss: 0.542\n",
      "Epoch 13: [20 / 21] loss: 0.526\n",
      "Evaluation accuracy in `test` split: 0.586\n",
      "Best model saved, epoch 13\n",
      "Start training in split train, Epoch: 14\n",
      "Epoch 14: [10 / 21] loss: 0.508\n",
      "Epoch 14: [20 / 21] loss: 0.491\n",
      "Evaluation accuracy in `test` split: 0.650\n",
      "Best model saved, epoch 14\n",
      "Start training in split train, Epoch: 15\n",
      "Epoch 15: [10 / 21] loss: 0.514\n",
      "Epoch 15: [20 / 21] loss: 0.490\n",
      "Evaluation accuracy in `test` split: 0.743\n",
      "Best model saved, epoch 15\n",
      "Start training in split train, Epoch: 16\n",
      "Epoch 16: [10 / 21] loss: 0.478\n",
      "Epoch 16: [20 / 21] loss: 0.487\n",
      "Evaluation accuracy in `test` split: 0.743\n",
      "Best model saved, epoch 16\n",
      "Start training in split train, Epoch: 17\n",
      "Epoch 17: [10 / 21] loss: 0.474\n",
      "Epoch 17: [20 / 21] loss: 0.437\n",
      "Evaluation accuracy in `test` split: 0.757\n",
      "Best model saved, epoch 17\n",
      "Start training in split train, Epoch: 18\n",
      "Epoch 18: [10 / 21] loss: 0.453\n",
      "Epoch 18: [20 / 21] loss: 0.467\n",
      "Evaluation accuracy in `test` split: 0.814\n",
      "Best model saved, epoch 18\n",
      "Start training in split train, Epoch: 19\n",
      "Epoch 19: [10 / 21] loss: 0.462\n",
      "Epoch 19: [20 / 21] loss: 0.491\n",
      "Evaluation accuracy in `test` split: 0.779\n",
      "Start training in split train, Epoch: 20\n",
      "Epoch 20: [10 / 21] loss: 0.457\n",
      "Epoch 20: [20 / 21] loss: 0.442\n",
      "Evaluation accuracy in `test` split: 0.800\n",
      "Start training in split train, Epoch: 21\n",
      "Epoch 21: [10 / 21] loss: 0.442\n",
      "Epoch 21: [20 / 21] loss: 0.435\n",
      "Evaluation accuracy in `test` split: 0.779\n",
      "Start training in split train, Epoch: 22\n",
      "Epoch 22: [10 / 21] loss: 0.446\n",
      "Epoch 22: [20 / 21] loss: 0.444\n",
      "Evaluation accuracy in `test` split: 0.807\n",
      "Start training in split train, Epoch: 23\n",
      "Epoch 23: [10 / 21] loss: 0.423\n",
      "Epoch 23: [20 / 21] loss: 0.405\n",
      "Evaluation accuracy in `test` split: 0.821\n",
      "Best model saved, epoch 23\n",
      "Start training in split train, Epoch: 24\n",
      "Epoch 24: [10 / 21] loss: 0.410\n",
      "Epoch 24: [20 / 21] loss: 0.420\n",
      "Evaluation accuracy in `test` split: 0.793\n",
      "Start training in split train, Epoch: 25\n",
      "Epoch 25: [10 / 21] loss: 0.414\n",
      "Epoch 25: [20 / 21] loss: 0.425\n",
      "Evaluation accuracy in `test` split: 0.800\n",
      "Start training in split train, Epoch: 26\n",
      "Epoch 26: [10 / 21] loss: 0.418\n",
      "Epoch 26: [20 / 21] loss: 0.397\n",
      "Evaluation accuracy in `test` split: 0.857\n",
      "Best model saved, epoch 26\n",
      "Start training in split train, Epoch: 27\n",
      "Epoch 27: [10 / 21] loss: 0.400\n",
      "Epoch 27: [20 / 21] loss: 0.388\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Best model saved, epoch 27\n",
      "Start training in split train, Epoch: 28\n",
      "Epoch 28: [10 / 21] loss: 0.381\n",
      "Epoch 28: [20 / 21] loss: 0.395\n",
      "Evaluation accuracy in `test` split: 0.836\n",
      "Start training in split train, Epoch: 29\n",
      "Epoch 29: [10 / 21] loss: 0.394\n",
      "Epoch 29: [20 / 21] loss: 0.400\n",
      "Evaluation accuracy in `test` split: 0.843\n",
      "Start training in split train, Epoch: 30\n",
      "Epoch 30: [10 / 21] loss: 0.380\n",
      "Epoch 30: [20 / 21] loss: 0.375\n",
      "Evaluation accuracy in `test` split: 0.879\n",
      "Start training in split train, Epoch: 31\n",
      "Epoch 31: [10 / 21] loss: 0.389\n",
      "Epoch 31: [20 / 21] loss: 0.408\n",
      "Evaluation accuracy in `test` split: 0.836\n",
      "Start training in split train, Epoch: 32\n",
      "Epoch 32: [10 / 21] loss: 0.379\n",
      "Epoch 32: [20 / 21] loss: 0.390\n",
      "Evaluation accuracy in `test` split: 0.857\n",
      "Start training in split train, Epoch: 33\n",
      "Epoch 33: [10 / 21] loss: 0.371\n",
      "Epoch 33: [20 / 21] loss: 0.362\n",
      "Evaluation accuracy in `test` split: 0.871\n",
      "Start training in split train, Epoch: 34\n",
      "Epoch 34: [10 / 21] loss: 0.380\n",
      "Epoch 34: [20 / 21] loss: 0.379\n",
      "Evaluation accuracy in `test` split: 0.879\n",
      "Start training in split train, Epoch: 35\n",
      "Epoch 35: [10 / 21] loss: 0.377\n",
      "Epoch 35: [20 / 21] loss: 0.379\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Best model saved, epoch 35\n",
      "Start training in split train, Epoch: 36\n",
      "Epoch 36: [10 / 21] loss: 0.361\n",
      "Epoch 36: [20 / 21] loss: 0.389\n",
      "Evaluation accuracy in `test` split: 0.871\n",
      "Start training in split train, Epoch: 37\n",
      "Epoch 37: [10 / 21] loss: 0.378\n",
      "Epoch 37: [20 / 21] loss: 0.376\n",
      "Evaluation accuracy in `test` split: 0.879\n",
      "Start training in split train, Epoch: 38\n",
      "Epoch 38: [10 / 21] loss: 0.374\n",
      "Epoch 38: [20 / 21] loss: 0.391\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 39\n",
      "Epoch 39: [10 / 21] loss: 0.385\n",
      "Epoch 39: [20 / 21] loss: 0.368\n",
      "Evaluation accuracy in `test` split: 0.879\n",
      "Start training in split train, Epoch: 40\n",
      "Epoch 40: [10 / 21] loss: 0.355\n",
      "Epoch 40: [20 / 21] loss: 0.367\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Best model saved, epoch 40\n",
      "Start training in split train, Epoch: 41\n",
      "Epoch 41: [10 / 21] loss: 0.373\n",
      "Epoch 41: [20 / 21] loss: 0.378\n",
      "Evaluation accuracy in `test` split: 0.879\n",
      "Start training in split train, Epoch: 42\n",
      "Epoch 42: [10 / 21] loss: 0.370\n",
      "Epoch 42: [20 / 21] loss: 0.353\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 43\n",
      "Epoch 43: [10 / 21] loss: 0.362\n",
      "Epoch 43: [20 / 21] loss: 0.349\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Best model saved, epoch 43\n",
      "Start training in split train, Epoch: 44\n",
      "Epoch 44: [10 / 21] loss: 0.341\n",
      "Epoch 44: [20 / 21] loss: 0.350\n",
      "Evaluation accuracy in `test` split: 0.886\n",
      "Start training in split train, Epoch: 45\n",
      "Epoch 45: [10 / 21] loss: 0.374\n",
      "Epoch 45: [20 / 21] loss: 0.354\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Best model saved, epoch 45\n",
      "Start training in split train, Epoch: 46\n",
      "Epoch 46: [10 / 21] loss: 0.344\n",
      "Epoch 46: [20 / 21] loss: 0.348\n",
      "Evaluation accuracy in `test` split: 0.886\n",
      "Start training in split train, Epoch: 47\n",
      "Epoch 47: [10 / 21] loss: 0.347\n",
      "Epoch 47: [20 / 21] loss: 0.339\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 48\n",
      "Epoch 48: [10 / 21] loss: 0.357\n",
      "Epoch 48: [20 / 21] loss: 0.360\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Best model saved, epoch 48\n",
      "Start training in split train, Epoch: 49\n",
      "Epoch 49: [10 / 21] loss: 0.349\n",
      "Epoch 49: [20 / 21] loss: 0.350\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 50\n",
      "Epoch 50: [10 / 21] loss: 0.360\n",
      "Epoch 50: [20 / 21] loss: 0.356\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 51\n",
      "Epoch 51: [10 / 21] loss: 0.350\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 51: [20 / 21] loss: 0.339\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 52\n",
      "Epoch 52: [10 / 21] loss: 0.343\n",
      "Epoch 52: [20 / 21] loss: 0.363\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 53\n",
      "Epoch 53: [10 / 21] loss: 0.342\n",
      "Epoch 53: [20 / 21] loss: 0.342\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 54\n",
      "Epoch 54: [10 / 21] loss: 0.341\n",
      "Epoch 54: [20 / 21] loss: 0.336\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 54\n",
      "Start training in split train, Epoch: 55\n",
      "Epoch 55: [10 / 21] loss: 0.319\n",
      "Epoch 55: [20 / 21] loss: 0.340\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 56\n",
      "Epoch 56: [10 / 21] loss: 0.325\n",
      "Epoch 56: [20 / 21] loss: 0.335\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 57\n",
      "Epoch 57: [10 / 21] loss: 0.315\n",
      "Epoch 57: [20 / 21] loss: 0.317\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 58\n",
      "Epoch 58: [10 / 21] loss: 0.325\n",
      "Epoch 58: [20 / 21] loss: 0.323\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 59\n",
      "Epoch 59: [10 / 21] loss: 0.317\n",
      "Epoch 59: [20 / 21] loss: 0.340\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 60\n",
      "Epoch 60: [10 / 21] loss: 0.367\n",
      "Epoch 60: [20 / 21] loss: 0.377\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 61\n",
      "Epoch 61: [10 / 21] loss: 0.335\n",
      "Epoch 61: [20 / 21] loss: 0.328\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 62\n",
      "Epoch 62: [10 / 21] loss: 0.330\n",
      "Epoch 62: [20 / 21] loss: 0.338\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 63\n",
      "Epoch 63: [10 / 21] loss: 0.335\n",
      "Epoch 63: [20 / 21] loss: 0.327\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 64\n",
      "Epoch 64: [10 / 21] loss: 0.318\n",
      "Epoch 64: [20 / 21] loss: 0.321\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 65\n",
      "Epoch 65: [10 / 21] loss: 0.335\n",
      "Epoch 65: [20 / 21] loss: 0.333\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 66\n",
      "Epoch 66: [10 / 21] loss: 0.324\n",
      "Epoch 66: [20 / 21] loss: 0.323\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 67\n",
      "Epoch 67: [10 / 21] loss: 0.315\n",
      "Epoch 67: [20 / 21] loss: 0.310\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 68\n",
      "Epoch 68: [10 / 21] loss: 0.317\n",
      "Epoch 68: [20 / 21] loss: 0.330\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 69\n",
      "Epoch 69: [10 / 21] loss: 0.314\n",
      "Epoch 69: [20 / 21] loss: 0.315\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 69\n",
      "Start training in split train, Epoch: 70\n",
      "Epoch 70: [10 / 21] loss: 0.302\n",
      "Epoch 70: [20 / 21] loss: 0.307\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 71\n",
      "Epoch 71: [10 / 21] loss: 0.291\n",
      "Epoch 71: [20 / 21] loss: 0.305\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 72\n",
      "Epoch 72: [10 / 21] loss: 0.307\n",
      "Epoch 72: [20 / 21] loss: 0.293\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 73\n",
      "Epoch 73: [10 / 21] loss: 0.312\n",
      "Epoch 73: [20 / 21] loss: 0.311\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 74\n",
      "Epoch 74: [10 / 21] loss: 0.309\n",
      "Epoch 74: [20 / 21] loss: 0.319\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 75\n",
      "Epoch 75: [10 / 21] loss: 0.318\n",
      "Epoch 75: [20 / 21] loss: 0.331\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 76\n",
      "Epoch 76: [10 / 21] loss: 0.311\n",
      "Epoch 76: [20 / 21] loss: 0.314\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 77\n",
      "Epoch 77: [10 / 21] loss: 0.311\n",
      "Epoch 77: [20 / 21] loss: 0.327\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 77\n",
      "Start training in split train, Epoch: 78\n",
      "Epoch 78: [10 / 21] loss: 0.343\n",
      "Epoch 78: [20 / 21] loss: 0.320\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 79\n",
      "Epoch 79: [10 / 21] loss: 0.332\n",
      "Epoch 79: [20 / 21] loss: 0.336\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 80\n",
      "Epoch 80: [10 / 21] loss: 0.322\n",
      "Epoch 80: [20 / 21] loss: 0.324\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 81\n",
      "Epoch 81: [10 / 21] loss: 0.322\n",
      "Epoch 81: [20 / 21] loss: 0.319\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 82\n",
      "Epoch 82: [10 / 21] loss: 0.310\n",
      "Epoch 82: [20 / 21] loss: 0.303\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 83\n",
      "Epoch 83: [10 / 21] loss: 0.304\n",
      "Epoch 83: [20 / 21] loss: 0.322\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 84\n",
      "Epoch 84: [10 / 21] loss: 0.317\n",
      "Epoch 84: [20 / 21] loss: 0.320\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 85\n",
      "Epoch 85: [10 / 21] loss: 0.319\n",
      "Epoch 85: [20 / 21] loss: 0.325\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 85\n",
      "Start training in split train, Epoch: 86\n",
      "Epoch 86: [10 / 21] loss: 0.316\n",
      "Epoch 86: [20 / 21] loss: 0.312\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 87\n",
      "Epoch 87: [10 / 21] loss: 0.311\n",
      "Epoch 87: [20 / 21] loss: 0.302\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 88\n",
      "Epoch 88: [10 / 21] loss: 0.318\n",
      "Epoch 88: [20 / 21] loss: 0.313\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 89\n",
      "Epoch 89: [10 / 21] loss: 0.309\n",
      "Epoch 89: [20 / 21] loss: 0.322\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 89\n",
      "Start training in split train, Epoch: 90\n",
      "Epoch 90: [10 / 21] loss: 0.305\n",
      "Epoch 90: [20 / 21] loss: 0.305\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 91\n",
      "Epoch 91: [10 / 21] loss: 0.318\n",
      "Epoch 91: [20 / 21] loss: 0.337\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 92\n",
      "Epoch 92: [10 / 21] loss: 0.349\n",
      "Epoch 92: [20 / 21] loss: 0.317\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 93\n",
      "Epoch 93: [10 / 21] loss: 0.311\n",
      "Epoch 93: [20 / 21] loss: 0.312\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 94\n",
      "Epoch 94: [10 / 21] loss: 0.307\n",
      "Epoch 94: [20 / 21] loss: 0.335\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 95\n",
      "Epoch 95: [10 / 21] loss: 0.319\n",
      "Epoch 95: [20 / 21] loss: 0.327\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 96\n",
      "Epoch 96: [10 / 21] loss: 0.302\n",
      "Epoch 96: [20 / 21] loss: 0.328\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 97\n",
      "Epoch 97: [10 / 21] loss: 0.320\n",
      "Epoch 97: [20 / 21] loss: 0.329\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 98\n",
      "Epoch 98: [10 / 21] loss: 0.314\n",
      "Epoch 98: [20 / 21] loss: 0.322\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 99\n",
      "Epoch 99: [10 / 21] loss: 0.321\n",
      "Epoch 99: [20 / 21] loss: 0.305\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 100\n",
      "Epoch 100: [10 / 21] loss: 0.308\n",
      "Epoch 100: [20 / 21] loss: 0.307\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 101\n",
      "Epoch 101: [10 / 21] loss: 0.295\n",
      "Epoch 101: [20 / 21] loss: 0.300\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 102\n",
      "Epoch 102: [10 / 21] loss: 0.297\n",
      "Epoch 102: [20 / 21] loss: 0.292\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 103\n",
      "Epoch 103: [10 / 21] loss: 0.300\n",
      "Epoch 103: [20 / 21] loss: 0.294\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 104\n",
      "Epoch 104: [10 / 21] loss: 0.299\n",
      "Epoch 104: [20 / 21] loss: 0.282\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 105\n",
      "Epoch 105: [10 / 21] loss: 0.297\n",
      "Epoch 105: [20 / 21] loss: 0.287\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 106\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 106: [10 / 21] loss: 0.286\n",
      "Epoch 106: [20 / 21] loss: 0.285\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 107\n",
      "Epoch 107: [10 / 21] loss: 0.327\n",
      "Epoch 107: [20 / 21] loss: 0.321\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 108\n",
      "Epoch 108: [10 / 21] loss: 0.309\n",
      "Epoch 108: [20 / 21] loss: 0.311\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Best model saved, epoch 108\n",
      "Start training in split train, Epoch: 109\n",
      "Epoch 109: [10 / 21] loss: 0.299\n",
      "Epoch 109: [20 / 21] loss: 0.300\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 110\n",
      "Epoch 110: [10 / 21] loss: 0.295\n",
      "Epoch 110: [20 / 21] loss: 0.300\n",
      "Evaluation accuracy in `test` split: 0.950\n",
      "Best model saved, epoch 110\n",
      "Start training in split train, Epoch: 111\n",
      "Epoch 111: [10 / 21] loss: 0.297\n",
      "Epoch 111: [20 / 21] loss: 0.299\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 112\n",
      "Epoch 112: [10 / 21] loss: 0.289\n",
      "Epoch 112: [20 / 21] loss: 0.342\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 113\n",
      "Epoch 113: [10 / 21] loss: 0.370\n",
      "Epoch 113: [20 / 21] loss: 0.344\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 114\n",
      "Epoch 114: [10 / 21] loss: 0.328\n",
      "Epoch 114: [20 / 21] loss: 0.313\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 115\n",
      "Epoch 115: [10 / 21] loss: 0.346\n",
      "Epoch 115: [20 / 21] loss: 0.351\n",
      "Evaluation accuracy in `test` split: 0.886\n",
      "Start training in split train, Epoch: 116\n",
      "Epoch 116: [10 / 21] loss: 0.330\n",
      "Epoch 116: [20 / 21] loss: 0.327\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 117\n",
      "Epoch 117: [10 / 21] loss: 0.314\n",
      "Epoch 117: [20 / 21] loss: 0.308\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Start training in split train, Epoch: 118\n",
      "Epoch 118: [10 / 21] loss: 0.291\n",
      "Epoch 118: [20 / 21] loss: 0.300\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 119\n",
      "Epoch 119: [10 / 21] loss: 0.285\n",
      "Epoch 119: [20 / 21] loss: 0.299\n",
      "Evaluation accuracy in `test` split: 0.921\n",
      "Start training in split train, Epoch: 120\n",
      "Epoch 120: [10 / 21] loss: 0.298\n",
      "Epoch 120: [20 / 21] loss: 0.302\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 121\n",
      "Epoch 121: [10 / 21] loss: 0.296\n",
      "Epoch 121: [20 / 21] loss: 0.291\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 122\n",
      "Epoch 122: [10 / 21] loss: 0.289\n",
      "Epoch 122: [20 / 21] loss: 0.294\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 123\n",
      "Epoch 123: [10 / 21] loss: 0.298\n",
      "Epoch 123: [20 / 21] loss: 0.283\n",
      "Evaluation accuracy in `test` split: 0.914\n",
      "Start training in split train, Epoch: 124\n",
      "Epoch 124: [10 / 21] loss: 0.291\n",
      "Epoch 124: [20 / 21] loss: 0.287\n",
      "Evaluation accuracy in `test` split: 0.907\n",
      "Start training in split train, Epoch: 125\n",
      "Epoch 125: [10 / 21] loss: 0.291\n",
      "Epoch 125: [20 / 21] loss: 0.293\n",
      "Evaluation accuracy in `test` split: 0.900\n",
      "Start training in split train, Epoch: 126\n",
      "Epoch 126: [10 / 21] loss: 0.306\n",
      "Epoch 126: [20 / 21] loss: 0.329\n",
      "Evaluation accuracy in `test` split: 0.893\n",
      "Start training in split train, Epoch: 127\n",
      "Epoch 127: [10 / 21] loss: 0.321\n",
      "Epoch 127: [20 / 21] loss: 0.320\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 128\n",
      "Epoch 128: [10 / 21] loss: 0.336\n",
      "Epoch 128: [20 / 21] loss: 0.327\n",
      "Evaluation accuracy in `test` split: 0.886\n",
      "Start training in split train, Epoch: 129\n",
      "Epoch 129: [10 / 21] loss: 0.331\n",
      "Epoch 129: [20 / 21] loss: 0.310\n",
      "Evaluation accuracy in `test` split: 0.936\n",
      "Start training in split train, Epoch: 130\n",
      "Epoch 130: [10 / 21] loss: 0.298\n",
      "Epoch 130: [20 / 21] loss: 0.292\n",
      "Evaluation accuracy in `test` split: 0.929\n",
      "Start training in split train, Epoch: 131\n",
      "Epoch 131: [10 / 21] loss: 0.307\n",
      "Epoch 131: [20 / 21] loss: 0.291\n",
      "Evaluation accuracy in `test` split: 0.943\n",
      "Train finish, best val score: 0.950\n",
      "Final test accuracy: 0.95\n"
     ]
    }
   ],
   "source": [
    "# run the model\n",
    "runner = Jobs(opt)\n",
    "max_score = runner.train()\n",
    "print(\"Train finish, best val score: {:.3f}\".format(max_score))\n",
    "runner.load_checkpoint(\"best.pth\")\n",
    "# runner.evaluate(\"test\")\n",
    "test_score = runner.translate()\n",
    "print('Final test accuracy: {}'.format(test_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92675548",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
